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For the last 3 years, more than 80% of the respondents to the Annual Phi Delta 
Kappa/Gallup Poll of the Public’s Attitudes Toward the Public Schools have 
stated that they would rather see a school s performance measure based upon 
“improvement shown by students ” than the “percentagepassing the test” (Rose 
& Gallup, 2007, p. 35). If this were to become the norm, the next question would 
be what “improvement” is significant? Educators need to understand “value- 
added ” if they are going to use “improvement ” to show that schools are improv¬ 
ing student achievement. 

Introduction 

H unt, Joseph, and Nuzzi (2002) summarize that “indicators continue 
to suggest that Catholic elementary and secondary school students, 
on average, outperform their public school counterparts” (p. 53). 
Unfortunately, this statement tells us little about the impact our schools have 
on influencing student achievement. Do Catholic schools have students with 
higher ability, greater parent support, and the other factors that influence a 
single point in time comparison of student achievement or are our students 
really achieving at a higher rate? 

As the federal government considers the reauthorization of No Child Left 
Behind (NCLB) legislation, there is a realization that emphasis on proficien¬ 
cy is not creating the desired level of increased student achievement. This 
creates both opportunities and challenges for Catholic schools. Under NCLB, 
parents of children in failing schools are now being given opportunities to 
choose a “better” school for their children. Whether Catholic schools fol¬ 
low state standards or not, the challenge is to provide evidence that Catholic 
schools are better able to maximize student achievement. 

The federal government is in the process of piloting the impact of a growth 
model in 9 states, requiring schools to demonstrate necessary proficiency and 
increased learning for all students in order to meet the goals of NCLB. The 
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common element among the approved states is that they use a growth model 
approach or value-added. Value-added models were first applied to school 
analyses by William Sanders at the University of Tennessee. The approach 
uses test data to measure “growth,” or “value-added,” meaning a student’s 
improvement from one year to the next. The results are then aggregated at the 
teacher, grade, and school levels. 

Past reporting of standardized test achievement results has concentrated 
on comparing buildings and systems: the student scores in one school to an¬ 
other school, Catholic schools to public schools, states to other states, and 
other types of comparisons. Previous to value-added, a school or a system 
could look good on average, but might not meet the needs of many indi¬ 
vidual students because comparing averages does not indicate if a student is 
achieving growth from one year to the next. The focus of value-added “for 
measuring student achievement appeals to administrators and policymakers 
at all levels of education because they quantify the gains that students make 
from one school year to the next rather than relying on reporting percentages 
of students who get passing or proficient scores at the end of the school year” 
(Viadero, 2008, p. 12). 

In 1999 the Archdiocese of Indianapolis, as part of a larger project, be¬ 
came interested in teacher improvement and began using value-added as¬ 
sessment for teacher performance-pay. The archdiocese received a school 
improvement grant from Lilly Endowment Incorporated and with matching 
funds raised by an archdiocesan corporate and business campaign, the Office 
of Catholic Education implemented many school improvement initiatives, 
one of which was the Teacher Advancement Program (TAP) developed by 
the Milken Family Foundation, which is “dedicated to attracting, develop¬ 
ing, motivating and retaining high-caliber educators in order to raise achieve¬ 
ment levels for all students” (National Institute for Excellence in Teaching, 
2008, Mission Statement). TAP began in 4 pilot schools in the fall of 2002 
and expanded to 13 schools in the 2006-2007 school year. The performance- 
based pay amounts are calculated using a formula that includes scores from 
teachers’ evaluations, value-added gains made by the students in the teach¬ 
ers’ classrooms, and the school wide value-added gains. The archdiocese has 
used value-added analyses for 5 years to measure these gains and to calcu¬ 
late the performance pay for teachers in 13 TAP schools. There have been 
four performance-based bonuses awarded to archdiocese TAP teachers and 
principals partially based on these value-added gains. In using the data it be¬ 
came apparent that it also provides the archdiocese with a tremendous tool 
for examining student achievement and using data to drive school improve¬ 
ment decision making. Beginning in February 2007, the Office of Catholic 
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Education had value-added reports calculated on all of our students in order 
to assist schools in improving their programs and making data-driven deci¬ 
sions. The reports were calculated on over 13,000 students and in 69 archdi¬ 
ocesan schools, which include urban, suburban, and rural demographics. 

Value-added assessment can demonstrate the impact Catholic schools 
have on student achievement. The purpose of this article is to explain 
value-added and how it is different from comparing student achievement as 
a single point in time on an annual basis; how schools in the Archdiocese of 
Indianapolis use value-added analysis to improve teaching and learning; and 
why it should be used. 

Value-Added and How It Is Different 
from Single Point in Time Achievement Data 

Value-added is a statistical method for computing the gains or progress stu¬ 
dents make in a pretest-to-posttest scenario (e.g., from one testing cycle to the 
next). If only a single point of measurement is used versus a pretest-to-post- 
test measurement, then there is no determination upon the amount of gains a 
student has made during a specified time frame. These value-added gains can 
be attributed to the school’s impact on student learning, because they are not 
affected by external characteristics, such as students’ family or neighborhood. 
These gains tell schools the impact they have on students who attend their 
institutions regardless of students’ level of achievement or measured profi¬ 
ciency. By using statistical models that combine the gains of numerous stu¬ 
dents, one can calculate the gains attributable to classroom teachers, schools, 
districts, and even entire states. 

In previous proficiency models, student achievement data is treated as 
an annual event because it is not li nk ed back to past student achievement. 
Through more than 20 years of research, Sanders and Rivers (1996) found 
that classroom teachers are the most important factor in achieving gains 
in student achievement. But Jencks and his colleagues (1972) found race, 
poverty, parental education, family structure, family interaction, parenting 
skills, and neighborhood factors account for 80 to 90% of that achievement. 
Jencks concluded, “the character of a school’s output depends largely on a 
single input, namely the characteristics of the entering children” (p. 256). 
According to David Berliner (2006), “if the educational opportunities avail¬ 
able to White students in our public schools were made available to all our 
students, the US would have been the 7th highest scoring nation in math¬ 
ematics, 2nd highest scoring nation in reading, and the 4th highest scoring 
nation in science” (p. 969). Relying on single point in time standardized 
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tests, therefore, reflects more the students’ background than the schools’ im¬ 
pact. Furthermore, if we only look at whether a student is above or below a 
set of proficiency expectations, then we miss the progress that the student is 
making over time. Figure 1 is an example of achievement for two students. 
Looking only at student proficiency, one can assume that student B was per¬ 
forming better than student A until the 9th grade when their performance lev¬ 
el is the same. At the 10th grade, student A is performing better than student 
B. In this scenario if individual student growth or progress is not monitored, 
then increases for the achievement of student A and the decline for student B 
over the years would have been missed. 



♦ Student A ■ Student B IN AYP Proficiencey 
Figure 1. Sample student achievement over time. 


How Value-Added Works 

In order to determine effectively the impact of the school or teacher on a 
student’s learning, individual student achievement data must be matched 
from one year to the next, then an expected level of growth is calculated 
based on the sample. The sample can consist of a district, multiple districts, 
a state, or even a nation. There are several needed conditions to compute 
value-added estimates: 

1. Each student must have two consecutive years of test data from a reliable 
and valid test (3 years is ideal) which is correlated to expected standards for 
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the student. Remember, value-added analysis is the longitudinal growth in 
achievement of the same set of students over time. 

2. Each student’s test scores must be linked to his or her teacher. When more 
than one teacher teaches a student, as is the case in most middle and high 
schools, teachers need to be linked to the student’s achievement in the subject 
area they teach. 

3. Districts and schools must use tests with sufficient vertical range between the 
scores from one year to the next to measure growth (any state test or norm- 
referenced test meets this criteria). 

Value-added statistics use mathematical modeling techniques to make the 
results more accurate and reduce the error associated with test scores. Value- 
added gains are estimations, not exact calculations, of the changes from the 
pre- and post-assessments for the students in question. Estimates are used 
because often there is data missing due to absence during testing, change in 
schools, and other various reasons. The value-added approach uses statisti¬ 
cal models to account for missing data to make precise estimates of student 
gains. In addition to computing these for districts and schools, these estimates 
can be computed on other variables, including race, gender, free lunch eligi¬ 
bility, Limited English Proficiency status, special education status, and oth¬ 
ers. These student-level variables can show how these variances impact the 
changes in student achievement. 

To compute a value-added estimate, it is best to assemble a large and 
diverse reference group of students and calculate their gains to serve as the 
context in which one interprets all gains. This group may be all the students 
in a school, district, state, or a sample of students across the nation, as is the 
case for norm-referenced tests. The size, scope, and geography of the refer¬ 
ence group will affect value-added estimates. If a reference group only has 10 
schools in it all from the same district, it is far less diverse and representative 
than a reference group of 1,000 schools from across the state. To reiterate, 
the reference group provides a context for comparison, so when the context 
changes, so do the estimates. 

How the Archdiocese of Indianapolis Uses Value-Added Results 

The Archdiocese of Indianapolis has received value-added reports for the last 
5 years. The analyses were first conducted by the Education Value Added 
Assessment System (EVAAS) Institute in North Carolina developed by 
William Sanders using the Terra Nova test and then the Indiana Statewide 
Test of Educational Progress (ISTEP). The diocese then contracted out its 
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value-added assessment with Teaching Doctors using its value-added analysis 
network software. The archdiocese found the Teaching Doctors ’ Value-Added 
Analysis Network’s reports were easier to understand and use. In addition, 
their reports explain how the calculations are determined (see Thum, 2003). 

Gain reports can be developed for all tested subjects (language arts, math¬ 
ematics, science, etc.), and for each subtest within a subject (e.g., reading 
comprehension, literature response, writing, etc.), so districts and schools can 
make more focused instructional and planning decisions. Figure 2 is an ex¬ 
ample of how gains are reported to teachers and administrators using reports 
from Teaching Doctors. 


Low Range 


Average % Gain 

High Range 



Average 

Gain 

Test/Subtest 

(-30) 

Mathematics 

(22) 

Computation 

(11) 

Geometry 

(-1) 

Measurement 

(-8) 

Number Sense 


-100 0 100 


Figure 2. Value-added gain report for a school on mathematics. 


A zero (0) percent gain means that the district, school, or teacher’s stu¬ 
dents gained no more or less than the representative sample. The bar is white 
when the average gain is above 0 and gray when it is below. The length of 
the bar represents all the scores that fall within a 75% certainty level of the 
mean reported gain. This provides an estimated gain of improvement based 
upon a range of scores that is not distorted by the extreme scores either side 
of the mean score. The center of the white or gray bars represent the average 
percent gain above the district average; the ends of the bars show the spread 
or confidence bands around the gain. For math, this example shows that the 
school’s average percent gain was -30, or 30% less than the representative 
gain. The range of the gain had a low of -70 and a high of -10. The final is¬ 
sue is how the interpretation of reports are used to change practices in our 
schools. 

Value-added reports are made available to archdiocesan schools in an 
electronic web format. This includes district-, school-, and grade-level reports 
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for all our Catholic schools in the archdiocese. Administrators and teachers 
review the reports and use the data to inform and improve instruction. 

The district-level chart from Teaching Doctors below demonstrates each 
school’s mathematics gains. Each bar represents one school. White bars are 
positive gains, gray bars are negative gains. School names are removed to 
maintain anonymity, but one can see from this report how one could look 
across all schools, and quickly assess their gains in the tested subject. The dis¬ 
trict report had 15 tests and subtests. Only mathematics gains are shown here. 



70 

65 


40 

35 

30 


-70 


Figure 3. Value-added gain report for district by school. 

To have significant positive gains, the school range of estimates had to be 
above the 0 line. The estimated average gains are based upon all students in 
the Archdiocese of Indiana schools in grades 3 through 10, which is slightly 
more than 13,000 students, who took the Indiana standards-based assessment 
(ISTEP). Any range of scores touching the 0 line would be average gains 
whether the bar was white or gray, and ranges of scores below the 0 line 
are negative. The smaller the range of scores shown in the bar around this 
mean (length of the bar for each school), the greater the consistency of the 
school on impacting the growth of students. In the archdiocese, 18 schools 
had value-added gains. 

Teaching Doctors reports are further disaggregated by creating school- 
and grade-level reports. Figure 4 displays a school level report by grade level 
for the two subject areas—English language arts and mathematics. English 
language arts has 6 subtests, and mathematics has 7 subtests. 

In the example for English language arts, a fourth grade-level report for 
one school, the highest gain score is in reading vocabulary, which has a val¬ 
ue-added range above the 0 line. The lowest is in reading comprehension, 
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Figure 4. Fourth grade value-added report by subject area. 


which has a mean below the 0 line, but the total range for English language 
arts is not significantly different because the bar touches the 0 line, which 
means student achievement gains at the subject level is not different than 
average gains estimated for the total population taking the standards-based 
assessment at this grade level. Therefore, value-added gains had not been 
achieved for this school in English language arts. All of the mathematics 
scores are positive and above the 0 line and are value-added gains. Similar 
reports are created at the individual grade levels to analyze student growth 
further. 

Making interpretations based on these value-added reports is a rela¬ 
tively straightforward process. In addition to viewing value-added reports 
by tested subjects at the district, school, grade, and teacher levels, further 
sub group reports are available by ethnicity; free and reduced lunch; high, 
middle, and low achievers; English language learners; special education and 
other student populations. This information on specific populations will al¬ 
low school leadership teams and teachers to plan for the specific needs of 
student populations. It must be kept in mind that the purpose of attaining 
value-added reports is to help administrators and teachers make informed 
and accurate decisions that can significantly impact classroom instruction 
and learning. Value-added analyses can assist schools in prioritizing when 
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creating improvement plans, because they provide data to inform responses 
to the following questions: 

1. On how many areas for improvement should we focus? 

2. Do the areas require a district-, school-, grade-, or teacher-level focus? 

3. What is a reasonable target for improvement in each area? 

For each area in need of improvement that a school identifies, it is im¬ 
portant also to analyze the reasons behind the results. One area for analy¬ 
sis of reasons is curriculum alignment and breadth of instructional content. 
Students must be provided opportunity to master the content standards on 
which they will be assessed. This assumption is based upon the belief that 
the assessments are aligned with the content standards that students should 
posess. This alignment should not be taken for granted, but should be moni¬ 
tored as value-added results become a part of curriculum alignment. Another 
area for analysis is instructional and assessment strategies. Are the students 
being taught effectively, are groups of students getting the content and learn¬ 
ing strategies they need to be successful? Finally, are time allocations ad¬ 
equate? How much teaching time is devoted to this area and is the use of time 
efficient and effective? 

Why Schools Should Use Value-Added 

Value-added reports should be used in making decisions about our improve¬ 
ment efforts in schools because now there is a measure of the expected level 
of improvement, not just a guess that the improvement is adequate. These es¬ 
timated gain scores are a more accurate assessment of student growth because 
they provide a pre- and post-assessment for each student, which eliminates 
the assessment error caused by single point in time comparisons with missing 
data for students who did not take both assessments. The value-added data 
allows for decision making based on whether students are making gains in 
specific areas to determine priorities in regard to professional development 
and generates data to inform responses to questions such as: What are the 
new strategies, methods, approaches, and materials needed to improve teach¬ 
ing and student learning? How should schools and dioceses decide on the 
need and allocation of current resources (texts, computers/software, specialist 
teachers, after school programs, etc.)? What are the needs of different popu¬ 
lations of students and what is the best way to reach them (ELL versus non- 
ELL, high versus low achievers, free lunch versus not free, minority versus 
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majority)? It is a matter of Catholic social teaching; meeting each student’s 
needs is critical to each student’s future success. 

Value-added analyses provide administrators, teachers, and schools with 
talking points to fashion school improvement efforts. They measure school 
and teacher impact on students’ learning, not family and neighborhood im¬ 
pact. Value-added results help schools more accurately determine where they 
are, and aid them in crafting informed plans for where they want to go this 
year and in subsequent years. In the end, it will allow the school to determine 
whether they made a year’s worth of value-added growth in student achieve¬ 
ment. It is what every child deserves. 

Concerns about Value-Added 

As with any model there are proponents and opponents. Proponents argue, 
as we do, that without a sophisticated method of comparing student pre- and 
post-achievement, real changes in student achievement go unnoticed, unmea¬ 
sured, and unexamined. Opponents criticize that value-added “has flaws that 
must be addressed...[including] a shortage of external reviews and validity 
studies of the model, its insufficient user-friendliness, and methodological is¬ 
sues about missing data, regression to the mean, and student background vari¬ 
ables” (Amerin-Beardsley, 2008, p. 65). It is beyond the scope of this paper 
to settle this debate, rather we argue that in our experience, value-added has 
allowed us to identify more clearly areas where students need to improve and 
determine whether student achievement has improved after implementing in¬ 
terventions to address those areas. We will continue to monitor our decisions 
and their impact based on the use of value-added data. 

Conclusion 

The need for Catholic schools to validate that they make significant (value- 
added) gains on student achievement is great. As stated in the Fordham B. 
Institute report Who Will Save America’s Urban Catholic Schools (Hamilton, 
2008), since 1990 over 1,300 Catholic schools have closed, displacing more 
than 300,000 students into public schools. If this trend is going to be reversed, 
Catholic schools need to overcome the “nostalgia and face these problems 
head-on” (p. 6). One of the recommendations is to “promote efforts to col¬ 
lect data, foster transparency and astutely ‘market’ Catholic schools” (p. 1). 
In the Fordham study, they could not cite one Catholic school system that 
would have been willing to be that “transparent.” Value-added promotes that 
transparency. If we are going to delve deeper in trying to demonstrate that 
our Catholic schools do make a difference in improving individual student 
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achievement, then we need more sophisticated methods of determining 
growth to ensure that we are not missing the real reason for growth. 

There are those who would argue that we really cannot measure student 
growth. Furthermore, there would be those who would state that this does not 
apply to Catholic schools. In this era of increased accountability and greater 
parental choice in selecting the school their child attends, we need to be think¬ 
ing about how we can prove that our schools make a difference in maximizing 
student achievement. 
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Questions about value-added? 

Log on to: http://valueadded3.teachingdoctors.com 
User name: guest@mail.com 
Password: Guest 
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